System and method for estimating daytime visibility

ABSTRACT

A system for estimating daytime visibility conditions includes a camera configured to capture a video sequence over a period time, an odometry sensor configured to monitor motion of the camera, and a video processing unit in communication with the camera and with the odometry sensor. The video processing unit is configured to receive the video sequence from the camera, detect an object within the video sequence, estimate a distance between the camera and the object for each frame of the video sequence, determine a mean perceived brightness of the object for each image frame, and estimate an ambient visibility using the estimated distance and mean perceived brightness of the object.

TECHNICAL FIELD

The present invention relates generally to systems for estimatingdaytime visibility conditions, such as may exist outside of anautomotive vehicle.

BACKGROUND

Visibility is typically a measure of the distance at which an object maybe clearly discerned. Visibility may be greatly reduced by variousenvironmental factors such as haze, fog, rain, and/or dust/sand. Theseenvironmental conditions may impair the ability of an operator of amotor vehicle to safely navigate even the most familiar of roads. Inparticular, such visibility conditions affect the maximum distance inwhich a vehicle operator can observe obstacles, pedestrians, and othervehicles.

SUMMARY

A system for estimating daytime visibility conditions includes a cameraconfigured to capture a video sequence over a period of time, anodometry sensor configured to monitor motion of the camera, and a videoprocessing unit in communication with the camera and with the odometrysensor.

The video processing unit may be configured to receive the videosequence from the camera, detect an object within the video sequence,estimate a distance between the camera and the object for each imageframe of the video sequence, determine a mean perceived brightness ofthe object for each image frame, and estimate an ambient visibilityusing the estimated distances and mean perceived brightness values ofthe object.

In one configuration, the video processing unit may estimate the ambientvisibility by assembling a plurality of point pairs, with each pointpair including an estimated distance of the object and a determined meanperceived brightness of the object. Each point pair may correspond tothe object in a different one of the plurality of image frames of thevideo sequence. Once the point pairs are assembled, the video processingunit may fit a curve to the plurality of point pairs, where the curvemay be based on a range-dependent object brightness model. From thefitted curve, a scattering coefficient may be determined to approximatethe ambient visibility.

The video processing unit may detect an object within the video sequenceby first detecting an image region having defined boundaries (i.e.,defined within a particular tolerance); and by examining the motion ofthe image region between at least two frames of the video sequence formotion of the object within the frames. The video processing unit maythen estimate the distance between the camera and the object usingdifferences in the image location of the object between at least two ofthe plurality of image frames, together with data from the odometrysensor (e.g., using triangulation techniques).

In a further embodiment, the video processing unit may be configured tocompare the estimate of ambient visibility to a threshold, and toperform a control action if the estimate exceeds the threshold. Forexample, the system may include a light source, such as an indicatorlight, or a fog light/headlamp, and the control action may includeilluminating the light source. In another embodiment, the system may beincluded with an automotive vehicle, and the control action may includelimiting the maximum speed of an engine of the vehicle. Furthermore, theestimate of ambient visibility may be used to weight various sensorinputs in a composite sensor fusion.

Similarly, a method of estimating an ambient visibility may includeacquiring a video sequence from a camera associated with a movingvehicle, wherein the video sequence includes a plurality of imageframes, detecting an object within the video sequence, estimating adistance between the camera and the object for each of the plurality ofimage frames, determining a mean perceived brightness of the object foreach of the plurality of image frames, and estimating an ambientvisibility using the estimated distance and mean perceived brightness ofthe object across the plurality of image frames. In a furtherembodiment, the method may additionally include comparing the estimateof ambient visibility to a threshold, and performing a control action ifthe estimate exceeds the threshold.

In one configuration, the step of estimating an ambient visibility mayinclude assembling a plurality of point pairs, where each point pairincludes an estimated distance of the object and a determined meanperceived brightness of the object. Each point pair may furthercorrespond to the object in a different respective one of the pluralityof image frames. The estimating may further involve fitting a curve tothe plurality of point pairs, which may be based on a range-dependentobject brightness model. From the curve, the system may then determine ascattering coefficient that may approximate the ambient visibility.Additionally, detecting an object within the video sequence may includedetecting an image region having defined boundaries, and examining themotion of the image region between at least two of the plurality ofimage frames for motion of the object within the frame.

Additionally, the method may include detecting a plurality of objectswithin the video sequence, estimating a distance between the camera andeach of the plurality of objects, determining a mean perceivedbrightness of each object, and determining a scattering coefficient fromthe estimated distances and mean perceived brightnesses.

The above features and advantages and other features and advantages ofthe present invention are readily apparent from the following detaileddescription of the best modes for carrying out the invention when takenin connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a vehicle including a system forestimating daytime visibility.

FIG. 2 is a schematic illustration of a video sequence captured by amoving camera over a period of time.

FIG. 3 is a schematic perspective illustration of a vehicle moving alonga path of travel and capturing a video sequence such as depicted in FIG.2.

FIG. 4 is a schematic block diagram of a video processing unitconfigured for estimating daytime visibility.

FIG. 5 is a schematic graph of the average brightness of an object at aplurality of estimated distances.

DETAILED DESCRIPTION

Referring to the drawings, wherein like reference numerals are used toidentify like or identical components in the various views, FIG. 1schematically illustrates a vehicle 10 having a video processing unit 12in communication with a forward-looking camera 14 and an odometry sensor16. As will be explained in greater detail below, the video processingunit 12 may estimate the visibility conditions outside the vehicle 10using the perceived brightness of one or more apparently stationaryobjects, as viewed by the camera 14, together with motion data receivedfrom the odometry sensor 16. This visibility estimation may subsequentlybe used to modify the behavior of different vehicle systems, such as forexample, the engine/powertrain 20 or headlights 22.

The video processing unit 12 may be embodied as one or multiple digitalcomputers or data processing devices, each having one or moremicroprocessors or central processing units (CPU), read only memory(ROM), random access memory (RAM), electrically-erasable programmableread only memory (EEPROM), a high-speed clock, analog-to-digital (A/D)circuitry, digital-to-analog (D/A) circuitry, input/output (I/O)circuitry, power electronics/transformers, and/or signal conditioningand buffering electronics. The individual control/processing routinesresident in the video processing unit 12 or readily accessible therebymay be stored in ROM or other suitable tangible memory locations and/ormemory devices, and may be automatically executed by associated hardwarecomponents of the video processing unit 12 to provide the respectiveprocessing functionality.

The video processing unit 12 may receive an analog or digital datastream 24 from one or more cameras 14 that are respectively configuredto capture video imagery within a particular field of view 26. The oneor more cameras 14 may each respectively include one or more lensesand/or filters adapted to receive and/or shape light from within thefield of view 26 onto an image sensor. The image sensor may include, forexample, one or more charge-coupled devices (CCDs) configured to convertlight energy into a digital signal. The one or more cameras 14 may bepositioned in any suitable orientation/alignment with the vehicle 10,provided that they may reasonably view one or more objects locatedadjacent to the path of travel (e.g., on the side of the road). In oneconfiguration, the camera 14 may be disposed within the front or reargrille of the vehicle 10. In another configuration, the camera 14 may bedisposed within one of the windshields of the vehicle 10 and oriented ina generally forward (or backward) facing direction (e.g., on aforward-facing surface of the rear-view mirror).

As generally illustrated in FIGS. 2-3, over time 30, a camera 14 maycapture a video sequence 31 that may include a plurality of discretetwo-dimensional images or image frames (e.g. image frames 32, 34, 36).Assuming the vehicle 10 is in motion, as generally represented in FIG.3, each frame 32, 34, 36 within the video sequence 31 may correspond tothe view of the camera 14 at a different respective position 38, 40, 42along the path of travel 44. Each image frame may then be separatelypassed from the camera 14 to the video processing unit 12 via the datastream 24, where it may be analyzed and/or processed to estimate theambient visability.

As schematically illustrated in FIG. 4, the video processing unit 12 mayinclude various processing modules that may receive the video datastream 24, and may ultimately output a control signal 46. The controlsignal 46 may, in turn, be based on a computed estimate of the ambientvisibility 48. Each processing module may be embodied, for example, as asoftware routine or control algorithm, which may be executed by thevideo processing unit 12. The processing modules may include, forexample, an object detection module 50, an object tracking module 52, anobject range estimation module 54, a visibility estimation module 56,and a system control module 58.

Upon receiving the video data stream 24, the object detection module 50may analyze each respective frame for the presence of one or morediscrete objects or image regions. These objects may include visualartifacts/image regions with clearly defined edges, and that may move ina continuous manner from frame to frame.

After detecting the presence of the one or more objects (e.g., streetsign 60), the object tracking module 52 may then quantitativelydetermine/track the location 62 of each respective object within eachframe object. For each identified object, the location 62 within theframe and mean brightness 64 may be recorded on a frame-by-frame basis.The location information 62 may then be passed to the object rangeestimation module 54, and the mean brightness 64 may be directly passedto the visibility estimation module 56.

In the object range estimation module 54, the video processing unit 12may use the location data stream 62, indicating the varying location ofthe one or more objects within the various frames 62, together with data66 from the odometry sensor 16 to estimate an approximate distancebetween the vehicle 10/camera 14 and the object at each frame.

The odometry sensor 16 may, for example, include an angular speed and/orangular position sensor associated with one or more vehicle wheels 68,which may be configured to monitor the rotational speed/position of thewheels. Additionally, the odometry sensor 16 may include a steeringsensor configured to detect the steering angle of the wheels. In thismanner, the odometry sensor 16 may obtain a sufficient amount ofinformation to estimate any vehicle motion (i.e., translation and/orrotation) and/or motion of the coupled camera 14 during vehicleoperation.

In an alternate configuration, a global positioning system can beutilized to determine the absolute motion and/or odometry of the vehicle10/camera 14. As may be understood in the art, a global positioningsystem may utilize externally broadcasted signals (typically broadcastedvia geo-stationary satellites) to triangulate an absolute position.Likewise a digital compass may be used to obtain the heading/orientationof the vehicle 10 throughout the range of vehicle motion.

The object range estimation module 54 may use vision-based processingalgorithms, such as for example, Structure-From-Motion techniques, toestimate the egomotion of the camera 14 (and vehicle 10) relative to theone or more detected objects. From this motion analysis, the rangeestimation module 54 may estimate a distance between the camera 14 andeach respective object on a frame-by-frame basis. For example, asschematically illustrated in FIG. 3, the range estimation module 54 mayattempt to estimate distances 70, 72, 74, between the vehicle 10 and theobject 60, for respective frames 32, 34, 36. The range-estimationalgorithm may be further informed using the actual recorded motion ofthe vehicle 10 (i.e., odometry data 66), which may be temporallysynchronized with the video data stream 24.

If the perceived motion of the object does not behave as expected (i.e.,distance rays 70, 72, 74 do not coincide at a point) then the object maybe discounted or rejected as being non-stationary and/or erroneous.Likewise, if there are multiple tracked objects, the object rangeestimation module 54, may look for a consensus behavior, and may thendiscount or reject objects that are found to be outside of a confidenceband of the consensus. Referring again to FIG. 4, once computed, theestimated object distances (collectively, distances 76) may be passedfrom the range estimation module 54 to the visibility estimation module56.

The visibility estimation module 56 may attempt to estimate the ambientvisibility using a theory of range-dependent object brightness.According to this theory, as object moves farther away from theobserver, the object's perceived brightness may become attenuated due tothe scattering of its radiance by particles in the medium. Thisattenuation may cause the object to look darker to the observer as anincreasing function of distance.

While the atmospheric conditions tend to attenuate the perceivedbrightness of an object as a function of distance, the same conditionscan also cause the general level of background brightness to increase asa function of distance, i.e., due to ambient light scattering.Therefore, the general perceived brightness (I) of an object can beexpressed according to Equation 1, which is generally derived fromKoschmieder's Law.

I=R·e ^(−βd) +A·(1−e ^(−βd))  Equation 1

As provided in Equation 1, R represents the radiance of the object, Arepresents the amount of available ambient atmospheric light, drepresents the distance between the perceiver and the object, and βrepresents the scattering coefficient. The scattering coefficient is anon-negative value, where a coefficient of 0 represents ideal orinfinite visibility, while on foggy conditions the scatteringcoefficient will be higher.

For each captured frame, the visibility estimation module 56 maycorrelate an object's mean perceived brightness (I) with the respectiveestimate of the distance to that object (d). As generally illustrated inFIG. 5, these point pairs may be plotted on a graph 80, with distance 82on the horizontal axis, and perceived brightness 84 on the verticalaxis. As shown, point 86 may correspond to object 60 in frame 32, point88 may correspond to the same object 60 in frame 34, and point 90 maylikewise correspond to the object 60 in frame 36. A curve 92 may befitted to these points according to the function described inEquation 1. As such, asymptote 94 may represent A, and the intercept 96(i.e., d=0) may represent R.

While R may be an object-specific variable, A and β should apply equallyto any object identified within the frame/sequence 31. Therefore, in asituation where multiple objects are identified, A and β may be selectedsuch that a curve may be drawn with minimal error (e.g. RMS) between thepoint pairs and curve for each respective object. Once the optimal curveparameters are determined, the scattering coefficient (β) may serve asan estimate of the ambient visibility 48, which may then be output tothe system control module 58.

In the estimation technique described above, the mean brightness 64 maybe computed in a manner that is largely independent of the resolution ofthe object. For example, the mean brightness may be equally representedwhether the object occupies 100 pixels or 100 k pixels. This techniquepresents advantages over video analysis techniques that may be relianton color variations or on the contrast of the object.

As generally described above, the system control module 58 may receivethe estimate of the ambient visibility 48, and may make one or morecontrol decisions in response. More specifically, the system controlmodule 58 may compare the received scattering coefficient (β) to one ormore stored thresholds. If the scattering coefficient exceeds the one ormore thresholds, the module 58 may infer that the vehicle is operatingin sub-optimal driving conditions (i.e., visibility is poor) and mayalert the driver and/or adjust the operation of one or more vehiclesystems in response. For example, in one configuration, if thevisibility is poor, the system control module 58 may provide a controlsignal 46 to the engine/powertrain 20 to artificially limit the maximumspeed of the vehicle 10. In another configuration, the system controlmodule 58 may provide a control signal 46 to the headlights 22 toilluminate the fog lights, or to prevent the high-beam headlights fromilluminating. In still other configurations, the control signal 46 maybe operative to alter the behavior of steering systems, vehiclemonitoring/response systems, driver alert systems, and/or transmissionshift points. In another configuration, where inputs from severalsensors of different modalities are fused into a unified surroundperception system, the reliability weight of each sensor could beadjusted according to its known limitations and sensitivity to adversevisibility conditions (e.g. radar may be less sensitive to adversevisibility conditions than cameras, and thus should be afforded a higherconfidence weight in the fused perceived object map).

In addition to the above described system/method, other perceived sensordata may be captured and fused with the visual data to determine theambient visibility. For example, radar information may be lesssusceptible to ambient visibility conditions such as fog or haze.Therefore radar data may be obtained and fused with the camera data tobetter determine the true position of the one or more objects, which mayaid in better estimating the scattering coefficient to a greaterconfidence.

While the best modes for carrying out the invention have been describedin detail, those familiar with the art to which this invention relateswill recognize various alternative designs and embodiments forpracticing the invention within the scope of the appended claims. It isintended that all matter contained in the above description or shown inthe accompanying drawings shall be interpreted as illustrative only andnot as limiting.

1. A system for estimating daytime visibility conditions comprising: acamera configured to capture a video sequence over a period of time, thevideo sequence including a plurality of image frames; an odometry sensorconfigured to monitor motion of the camera and output odometry dataindicative of the monitored motion; a video processing unit incommunication with the camera and with the odometry sensor, the videoprocessing unit configured to: receive the video sequence from thecamera; detect an object within the video sequence; estimate a distancebetween the camera and the object for each of the plurality of imageframes; determine a mean perceived brightness of the object for each ofthe plurality of image frames; and estimate an ambient visibility usingthe estimated distance and mean perceived brightness of the objectacross the plurality of image frames.
 2. The system of claim 1, whereinthe video processing unit is further configured to compare the estimateof ambient visibility to a threshold, and to perform a control action ifthe estimate exceeds the threshold.
 3. The system of claim 2, furthercomprising a light source, and wherein the control action includesilluminating the light source.
 4. The system of claim 1, whereinestimating an ambient visibility includes: assembling a plurality ofpoint pairs, each point pair including an estimated distance of theobject and a determined mean perceived brightness of the object, andcorresponding to the object in a different respective one of theplurality of image frames; fitting a curve to the plurality of pointpairs, the curve based on a range-dependent object brightness model; anddetermining a scattering coefficient from the fitted curve.
 5. Thesystem of claim 1, wherein detecting an object within the video sequenceincludes: detecting an image region having defined boundaries; andexamining the motion of the image region between at least two of theplurality of image frames for motion of the object within the frame. 6.The system of claim 1, wherein the video processing unit is configuredto estimate the distance between the camera and the object using thevarying location of the object between at least two of the plurality ofimage frames, together with the received odometry data.
 7. The system ofclaim 6, wherein the video processing unit is further configured toestimate the distance between the camera and the object using astructure-from-motion processing technique.
 8. The system of claim 1,wherein the odometry sensor includes a wheel speed sensor and a steeringangle sensor.
 9. A method of estimating an ambient visibilitycomprising: acquiring a video sequence from a camera associated with amoving vehicle, the video sequence including a plurality of imageframes; detecting an object within the video sequence; estimating adistance between the camera and the object for each of the plurality ofimage frames; determining a mean perceived brightness of the object foreach of the plurality of image frames; and estimating an ambientvisibility using the estimated distance and mean perceived brightness ofthe object across the plurality of image frames.
 10. The method of claim9, further comprising: comparing the estimate of ambient visibility to athreshold; and performing a control action if the estimate exceeds thethreshold.
 11. The method of claim 10, wherein performing a controlaction includes illuminating a light source.
 12. The method of claim 10,wherein performing a control action includes speed-limiting an engine.13. The method of claim 9, wherein estimating an ambient visibilityincludes: assembling a plurality of point pairs, each point pairincluding an estimated distance of the object and a determined meanperceived brightness of the object, and corresponding to the object in adifferent respective one of the plurality of image frames; fitting acurve to the plurality of point pairs, the curve based on arange-dependent object brightness model; and determining a scatteringcoefficient from the fitted curve.
 14. The method of claim 9, whereindetecting an object within the video sequence includes: detecting animage region having defined boundaries; and examining the motion of theimage region between at least two of the plurality of image frames formotion of the object within the frame.
 15. The method of claim 9,further comprising receiving data indicative of the motion of thecamera; and wherein estimating the distance between the camera and theobject includes analyzing differences in the appearance of the objectbetween at least two of the plurality of image frames, in view of thereceived motion data.
 16. The method of claim 9, wherein the object is afirst object; and further comprising: detecting a second object withinthe video sequence; estimating a distance between the camera and thesecond object for each of the plurality of image frames; determining amean perceived brightness of the second object for each of the pluralityof image frames; and wherein estimating an ambient visibility includesdetermining a scattering coefficient from the estimated distances andmean perceived brightnesses of the first and second objects across theplurality of image frames.
 17. The method of claim 9, further comprisingfusing perceived radar information with the acquired video sequence. 18.The method of claim 9, further comprising adjusting the weighting of oneor more perception systems in a composite sensor fusion using theestimate of ambient visibility.